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ABSTRACT 



The practice of reforming failing schools through school 
reconstitution is one of the latest challenges for school system directors of 
evaluation and accountability. Failing schools are typically identified in 
terms of student test performance. However, evaluating the impact of 
reconstitution in terms of test score gains may not be an appropriate 
methodology for the short run. Instead, evaluators should consider multilevel 
evaluation models that account for those school input factors that were 
reconstituted, i.e., teaching staff and class size,. The position is taken 
that it is vital to delineate, obtain, and report information on teacher and 
class size variables in an evaluation of school reconstitution reform. This 
paper attempts to define the nature of the school reconstitution evaluation 
problem, relate teacher experience/training and class size to school reform, 
delineate operational definitions for obtaining standardized teacher and 
class size variables, and show some descriptive and correlation results for 
these variables. (Contains 3 tables and 16 references.) (Author/SLD) 
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Abstract: The practice of reforming failing schools through school reconstitution 
is one of the latest challenges for school system directors of evaluation and 
accountability. Failing schools are typically identified in terms of student test 
performance. However, evaluating the impact of reconstitution in terms of test 
score gains may not.be an appropriate methodology for the short run. Instead, 
evaluators should consider multilevel evaluation models which account for those 
school input factors which were reconstituted, i.e., teaching staff and class size. 
The authors take the position that it is vital to delineate, obtain and report 
information on teacher and class size variables in an evaluation of school 
reconstitution reform. This paper attempts to define the nature of the school 
reconstitution evaluation problem, relate teacher experience/training and class size 
to school reform, delineate operational definitions for obtaining standardized 
teacher and class size variables, and show some descriptive and correlation results 
for these variables. 



The practice of reforming failing schools through school reconstitution is one of the latest 
challenges for school district directors of evaluation and accountability. Because schools are 
targeted for reconstitution based on low test scores, increased academic achievement is the 
primary goal of the reconstitution reform. However, an evaluation design which relies on 
student test scores may not be the best method for assessing the impact caused by reconstitution. 

Employing case study methodology to examine the reconstitution impact on a school’s 
resources, governance methods, school-community relations, organizational infrastructure and 
instructional programs is an alternative design with many admirers (Finkelstein, et. al.,1998). 
Case study methodology can capture the ways in which schools have interpreted and reacted to 
reconstitution. This important information, however, is very costly to collect, frequently 
criticized as subjective, not amenable to shorthand quantitative presentations, and generally out 
of reach of most school district directors of evaluation. 



Paper presented at American Educational Research Association, 1999 Annual Meeting, Montreal, 
Canada, for Division H Symposium: Evaluating the Effectiveness of School Reconstitution 
4/23/99. 
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Another alternative is to broaden the focus of the evaluation on student test scores to 
include those input factors which are the primary result of the reconstitution effort, namely the 
teaching staff and class size. Following Stufflebeam’s CIPP model, this initial focus on program 
input evaluation will secure information needed to appraise the teacher and class size changes 
resulting from the school reconstitution. 2 The input evaluation task is to ascertain the nature of 
available capabilities resulting from the staffing reconstitution and determine the potential for 
achieving the needed student academic gains which identified the school for reform in the first 
place. The input evaluator’s task is to delineate those teacher and class variables which 
accurately reflect the reconstitution effort, obtain data which measure the delineated teacher and 
class variables, and provide analysis information needed for decisions regarding which 
instructional resources to use and in what manner they should be employed in order to promote 
achievement of the reconstitution objective identified, namely student academic gains. 

Educators and researchers know that the placement of better teachers in a school is one of 
the most influential determinants to student achievement that is under the reconstitution control 
of school officials (Phillips and Adcock, 1997; Wright, Horn, & Sanders, 1997). Reconstituting 
schools, however, is not necessarily the same thing as improving the teaching faculty or the 
teaching situation. It would not be reasonable, for example, to expect student achievement gains 
in reconstituted schools that do not accrue measurable improvements in the school’s teaching 
capacity or classroom teaching situation. Consequently, it is the position of the authors that a 
school district accountability system for evaluating the impact of a school reconstitution reform 
program should first measure inputs in terms of improvements in school teaching capacity and 
classroom teaching situations. 

From the perspective of the office of Research, Evaluation and Accountability (REA) in a 
large public school system that has reconstituted failing schools, this paper attempts to: 

• describe the nature of the school reconstitution program as implemented in the Prince 
George’s County Public Schools’ (PGCPS) system; 

• review the rationale for including teacher experience/training and class size indicators 
into an accountability system for evaluating school reconstitution reform; 

• provide operational definitions for teacher and class variables; and 

• show some descriptive and correlation results for these variables 



BACKGROUND: SCHOOL RECONSTITUTION IN PGCPS 

The schools in the Prince George’s County Public Schools’ (PGCPS) system are 
accountable to two agencies, State and Local, which may target them for reconstitution. Both 



' Daniel Stufflebeam and Egon Guba originated the CIPP Model which is an acronym representing 

the four types of evaluation this model identifies, namely, context evaluation, input evaluation, process evaluation, 
and product evaluation. 
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accountability systems share a primary emphasis on student test performance for their school 
reconstitution selection and evaluation process. The pressures on PGCPS from the State 
accountability program peaked in the late 1990s, when the State threatened to reconstitute 
schools if their performance did not improve. In the Spring of 1997, the district superintendent, 
in concert with the Board of Education of the Prince George’s County Public Schools’ (PGCPS) 
system initiated a local school reconstitution process for six of the system’s lowest performing 
schools entitled: “The 2 1st Century Schools Initiative. ” The reform “was created for the purpose 
of significantly improving the academic achievement of students enrolled in these schools.” The 
reform called for a near total reconstitution of the administration, instructional, and building 
support staff at these six schools. It is this local 2 1st Century Schools Initiative school 
reconstitution that is the focus of this paper. 

The primary selection criteria for failing school status was low and declining performance 
in State mandated tests. The Maryland State Department of Education’s (MSDE) accountability 
program measures school status differently at each school level (i.e., elementary, middle, and 
high school). At the elementary school level, the primary performance measure for school 
accountability status is calculated by aggregating third- and fifth-grade student test performance 
relative to a “satisfactory” standard on constructed performance assessments in six content areas 
(language usage, mathematics, reading, science, social studies, and writing) along with 
attendance. The middle school status is calculated by aggregating eighth grade test performance 
in the same six content areas, along with attendance and the middle school’s “graduates” 
performance on the ninth grade Maryland Functional Mathematics, Reading, and Writing Tests. 
The high school accountability status is calculated by aggregating student performance, in both 
the ninth and eleventh grades, on the Maryland Citizenship Test and the Functional Mathematics, 
Reading, and Writing Tests along with daily attendance and dropout data. 

Based upon this school performance accountability system, the 21st Century Schools 
Initiative identified six schools (four elementary and two middle schools) for reconstitution. The 
2 1 st Century Schools Initiative called for a complete overhaul of instructional staff at each of the 
sites. More specifically, it (1) recalled all teachers and administrators in each of the six schools; 
(2) required all staff that sought positions in the six 21 st Century Schools to apply (or reapply) 
and to be interviewed; and (3) guaranteed positions within the county for all teachers who had 
previously taught in the 21 st Century Schools. An extensive case study of the first year 2 1st 
Century Schools Initiative implementation, conducted by an Interdisciplinary Education Policy- 
Study Team, from the University of Maryland, reported that the initiative virtually “cleaned the 
slate” of staff in the six sites targeted for reconstitution (Finkelstein, et. al, 1998). 

The Superintendent explained the staff restructuring in a July 1, 1997 memorandum as a 
means to provide these six schools with a new group of teachers and new principals that “...will 
willingly ‘break the mold’ to pose unique and inventive solutions to problems; that they will be 
data-driven in their assessments of ‘what works;’ and that they will always take actions that are 
in the best interests of children.” While new staff were hired at all school sites, it is not clear 
whether additional positions were created or that additional dollars were spent on refilled staff 
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positions as a result of the 2 1st Century Schools Initiative. A case study investigation conducted 
by a University of Maryland Interdisciplinary Education Policy Study Team found that the 2 1st 
Century Schools sites did receive new resource allocations for staff development and monies 
structured to support educational programs and tuition supplements for teachers pursuing 
Master’s degrees, formal certification, and general education (Finkelstein, et al., 1998). 

While the benefits of staff development activities remains questionable in the literature, 
research continues to demonstrate that teacher experience and academic training prior to 
engagement in such instructional challenges as school reform practices is of vital importance. 
The next section presents a brief review of the literature on the relationship of teacher experience 
and academic training. 



TEACHER TRAINING/EXPERIENCE AND CLASS SIZE INDICATORS 

Research on class size and teachers has been particularly plagued by a variety of 
methodological and definition problems. Early research in these areas relied on macro-level 
data that had been aggregated at levels far from the classroom (e.g., district and state). More 
recent research that is defining class size in terms of classroom teachers and using value-added 
analysis methodology at the school and teacher levels is yielding robust results which reveal 
strong evidence of a teacher effect and mixed results on class size effect. 

Hanushek’s classic 1986 review of production function studies found that most indicator 
research includes three school inputs that are the largest determinants of instructional 
expenditures — teacher experience, teacher education, and class size. Ferguson’s 1991 district- 
level regression analysis study using almost 900 school districts in Texas linked student test 
performance in reading to teachers’ language skill. Ferguson’s analysis revealed that after 
teachers’ language skills, the next most important school characteristic in promoting children’s 
reading performance is teacher experience, followed by class size and the proportion of teachers 
with master’s degrees. In support of Hanushek’s earlier observations, Ferguson’s study 
demonstrated with empirical estimates at the school district level that teachers’ tested skill level, 
teacher years of experience, class size and teachers’ college training are important variables for 
empirical studies of “quality schooling” and “best practices.” 

The PGCPS school-level findings from value-added research with 120 elementary 
schools are consistent with Ferguson’s 1991 district-level regression analysis study. Using 
school-wide aggregated data, Phillips and Adcock (1996) and Adcock and Phillips (1997) found 
that elementary students’ test performance on a statewide accountability measure in reading was 
linked to the average college training level of classroom teaching staff and, to a much lesser 
degree, the average class size for a school. Modeling the district data as a two-level students- 
nested-within-school structure showed that the teacher college training level and class size input 
variables impacted students’ reading test scores. Moreover, the impact of college training level 
of the school’s teaching staff and average class size were independent of students’ family 
income, mobility and minority status. 
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More recently, Wright, Horn, and Sanders (1997) conducted a large-scale longitudinal 
teacher-level study as a component of the Tennessee Value-Added Assessment System 
(TVAAS). Sanders et al., found that teacher effects are the dominant factors affecting student 
academic gain and that class sizes have relatively little influence on academic gain. The 
February 18, 1998 edition of Education Week reports that Mendro and his team of researchers in 
the Dallas district have replicated and extended the Tennessee research work. They used a value- 
added hierarchical two-stage, two-level student-within-teacher analysis model that demonstrated 
strong links between students’ assessment performance and teachers’ measured skill level effects 
in a large-scale study of 1,500 teachers. The studies from the state of Tennessee and the Dallas 
school district represent the most extensive empirical application of value-added research 
methodology at the teacher-level in the country. 

In summary, the research continues to underscore the dramatic effects skilled teachers 
have on student performance, but the findings on class size is less impressive. Yet, one of the 
best known “common knowledge” ways to improve student learning is to reduce class size. As 
presented in this section, educational researchers have long sought to find evidence to support or 
refute this claim (Glass et al., 1982; Hanushek, 1986; Ferguson, 1991; and Word, 1990). As 
researchers increasingly use more robust methodology and standardize variable definitions for 
teachers and class size, the evidence reveals a class size effect, especially for the earlier grades. 
Most of the articles and studies emphasize that in order to experience student achievement gains, 
positive changes must occur in teaching practices along with the reduction in class size. In any 
regard, the research implications for school reconstitution clearly point to a need for providing 
decision makers with the input evaluation information on teacher and class size capability 
improvements which have resulted from the reform initiative. 



DATA DEFINITION FOR TARGETED INDICATORS 

The literature shows that teacher experience, teacher education and class size are 
important school level input variables for school reform evaluations. This is especially true for 
school reconstitution programs that target changes in teaching staff and class size as the primary 
instrument of reform. Assessing the impact of these school input variables is complicated by 
definitions, data processing and reporting issues. While most observers believe they know who a 
“teacher” is, and certainly could recognize a “class” when they are looking at one, it is quite 
another matter to extract such school input variables from school district databases for evaluation 
purposes. 

Having convincing evidence that school reconstitution evaluation needs to include 
teacher and class size input variables into the analysis is one thing; however, “doing the data” 
with sufficient scientific rigor to support multilevel analysis modeling is quite another. Data is 
expensive, and evaluation quality data is very expensive. Two highly readable first 
introductions for transforming school district legacy data into evaluation quality data is presented 
in papers by Haseltine, Adcock and Winkler (1998) and Adcock and Phillips (1997). 
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The National Center of Educational Statistics (NCES) has produced a seminal document 
for the standardization of school district data in the form of their 1991 publication on '‘Standards 
for Education Data Collection and Reporting” (SEDCAR). The SEDCAR provides a 
comprehensive guide for state and local education agencies for collecting, processing and 
reporting school system data elements and variables. Unfortunately, armed with even these 
excellent “how to” guides, Blank (1993) points out that evaluators still face many hurdles which 
impede the development of a system to collect, process and report target indicator data from the 
typical school district legacy data management system. 

Despite the difficulties posed by the typical school district legacy data systems, district 
directors of evaluation need to find a way to delineate, collect, process and report teacher and 
class input data in order to provide meaningful evaluation information on the impact of school 
reconstitution reform. Using the SEDCAR data processing standards and related student and 
staff handbooks as guides for data definitions, the Research, Evaluation and Accountability 
(REA) office of the Prince George’s County Public Schools (PGCPS) has developed the 
Research and Evaluation Assimilation Database (READ) support system for evaluation activities 
in the district (Adcock, Haseltine & Winkler, 1997). Generally speaking, READ is a relational 
database which provides detailed student achievement data together with context and input 
information at the various levels of students, classrooms, teachers and schools. Technically 
speaking, READ is a research office data mart 3 designed to provide Sufficient Statistical Matrix 
(SSM) data files that fulfill the data modeling requirements of multi-level analysis (e.g., HLM, 
SEM, AMOS, and LISREL). 

For the READ system, the data element definitions pertain to different units of measure, 
i.e., individual, classroom, school, program, and district. These data elements, and their 
concomitant definitions, also serve many cohort level studies through simple derivations and 
translations. 

For school reconstitution data analysis, the core classroom teacher is the primary unit of 
measure. The standardized READ definition for a core teacher is: 

A core teacher is a school-based, certified or provisionally certified classroom 
teacher employed on the last day of the school year and who has the assigned 
responsibility to provide students instruction and assign course grades in one or 
more of the core academic subject areas of language (including reading and 
English), mathematics, science, or social studies. 

The READ system core secondary school teacher is determined in a straightforward manner from 
individual student course records. Because PGCPS does not use a similar computerized course 
grading system at the elementary level, determining who are the elementary students’ core 
teachers is more problematic. Each Spring, REA administers a Student-Teacher-Subject Survey 
to all elementary schools in the district. This survey captures each elementary grade students’ 
teacher(s) for reading, mathematics, social studies and science from lists of students provided by 



3 A data mart is a departmental-sized data warehouse which contains a subset of an enterprise’s overall 
evaluation support data of interest to the particular department (e.g., research office). 
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REA. The students’ core teacher records are merged with personnel data to yield the follow data 
elements for analysis: 

Teacher college training is the college academic training index of the core 
teacher on a seven-point scale: l=Bachelors, 2=Bachelors+30 course credit hours 
(cch), 3=Masters/Equivalent, 4=Masters+15(cch), 5=Masters+30(cch), 
6=Masters+60(cch), 7=Doctorate. 

The teacher service years at the school is the total number of years that the core 
teacher was employed at the “event” school based on their most recent occurrence 
of continuous employment at that school. Only the last continuous belonging 
period was considered. 

The teacher service years in the system is calculated as the average total number 
of years that core teachers have been employed as teachers (provisional or 
certified) in the system based upon their most recent occurrence of continuous 
employment in the system. Only the last continuous belonging period was 
considered. 

The teacher years of experience is the total number of years of teaching credit 
assigned regardless of location (i.e., in or out of the district). These do not have to 
be continuous years of teaching employment. 

The teacher certification status is the instructional certification type achieved by 
the core teacher on a six-point scale: l=Provisional Without-Degree (Voc-Tech); 
2=Provisional With Degree; 3=Resident Teacher; 4=Standard Professional-I; 
5=Standard Professional-II; and 6=Advanced Professional. 

The teacher certification area is the discipline(s) or specific subject area(s) in 
which a core teacher has been certified to instruct based upon documented 
academic training and/or test performance. The core teacher READ record 
contains as many certification area fields as the teacher has areas/fields of 
certification. 

As developed in previous sections, a school’s class size statistic is important to the 
evaluation of school reconstitution reform. The READ system class size statistic is actually a 
“constructed class size” count, sometimes referred to as a student-core teacher ratio count in the 
literature. This statistic is calculated from the total number of students divided by the “school 
level count of core teachers.” The school level count of core teachers is itself a derived score 
based upon either the observed teacher count (via elementary Spring survey or secondary student 
course records) or the Pupil Accounting Class report, whichever count is the larger. This method 
assures that the READ school number core teacher count reflects the individual classroom 
instructional situation employed by each school site. 

In sum, the school’s constructed class size is the end-of-year student enrollment divided 
by the determined school number of core classroom teachers for each school. This student-core 
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teacher ratio calculation usually yields a larger class size number than the other class size 
calculations seen in the literature, but as Ferguson (1991) admits, the other calculations are prone 
to underestimation because they include more than core teachers in the denominator. The 
standardized READ definition for a school’s class size is: 

The class size is the total number of students enrolled on the last day of the school 
year divided by the derived school number of core teachers employed on the last 
day of the school year for each elementary school. 

Table 1 lists the student, teacher, and school level variables which may be considered 
fundamental to any evaluation of the impact of school reconstitution reform. Specifications for 
“student” and “teacher” are provided in order to anchor the list of school input factors that are the 
subject of this paper. Specifically, Table 1 presents the delineated variable definitions for 
student, teacher and school level variables, the variable coding schemes for individual records, 
the computational algorithms for summarizing school level scores, and related scaling parameters 
and conditions. It is the position of the authors that school district directors of evaluation should 
provide at least this level of specificity in the development of their evaluation data set to meet 
standards of reproducibility and lend confidence to the information yielded by their analysis. 
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Table 1 



Target Data for Evaluating School Reconstitution 
List of Student, Teacher and School Level Variables 



Variable 


Definition 


Comment 


Student 


A student is a person who is enrolled in 
PGCPS for at least one day, and has an 
enrollment record containing ah of the 
following: a valid student ED number, a 
first and last name, a valid race code, a 
valid gender code, a date of birth. 


“Student” is a Research, 
Evaluation and Assimilation 
Database (READ) warehouse 
system data element. For 
evaluation purposes, student 
is often conditionally defined 
in terms of the study in 
question (i.e., in terms of a 
year cohort, the student’s 
teacher, school, program, 
course, test, etc.). 


Teacher Level Variables 


Teacher 


A core teacher is a school-based, 
certified or provisional certified 
classroom teacher employed on the last 
day of the school year and who has the 
assigned responsibility to provide 
students instruction and assign course 
grades in one or more of the core 
academic subject areas of language 
(including reading and English), 
mathematics, science, or social studies. 


Core teacher is a READ 
system data element. 
Depending on the nature of an 
evaluation study, the number 
of core teachers for an entity 
(e.g., school, program, or year 
cohort) can be either observed 
or derived from other 
accounting sources. 


Teacher College 
Training 


The academic training index of the core 
teacher on a seven point scale: 
l=Bachelors, 2=Bachelors+30 course 
credit hours (cch), 3=Masters/Equivalent, 
4=Masters+ 1 5(cch), 5=Masters+30(cch), 
6=Masters+60(cch), 7=Doctorate. 


Data source is Personnel 
Office college training code 
for individual teachers. 
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Variable 


Definition 


Comment 


Teacher 

Certification 

Status 


The teacher certification status is the 

instructional certification type achieved 
by the core teacher on a six point scale: 
l=Provisional W/O-Degree (Voc-Tech); 
2=Provisional With-Degree; 

3=Resident Teacher; 

4=Standard Professional-I; 

5=Standard Professional-II; and 
6= Advanced Professional. 


Data source is Personnel 
Office college training code 
for individual teachers. 


Teacher 

Certification 

Area 


The teacher certification area is the 

discipline(s) or specific subject area(s) 
which a core teacher has been certified to 
instruct based upon documented 
academic training and/or test 
performance. The core teacher READ 
record contains as many certification area 
fields as the teacher has areas/fields of 
certification. 


Data source is Personnel 
Office college training code 
for individual teachers. 


Teacher Service 
Years School 


The total number of years that the 
individual core teacher was employed at 
the “event” school. Years of experience 
operationally defined as “belonged” to 
that school based upon their most recent 
occurrence of continuous employment in 
that school. 


Only the last continuous 
“belonging” period in the 
“event” school is considered. 
Data source is summarized 
Personnel Office records. An 
“event” school is location of 
interest to the evaluation study. 


Teacher Service 
Years in System 


The teacher service years in the system 

is calculated as the average total number 
of years that core teachers have been 
employed as teachers (provisional or 
certified) in the system based upon their 
most recent occurrence of continuous 
employment in the system. 


Data source is Personnel 
Office for individual teachers. 
Only the last continuous 
belonging period is considered 


Teacher Service 
Experience 


The teacher years of experience is the 
recorded total number of years of 
teaching credit assigned for teaching 
regardless of location. 


Data source is Personnel 
Office. The sum of years 
includes all teaching 
experience (provisional or 
certified) in any approved 
educational setting (private or 
public). 
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Variable 


Definition 


Comment 


School Level Variables 




School Level 
Student Count 


The school level student count is the 

number of enrollees belonging to a 
school site on a given day, usually on the 
last day of the school year or at some 
specified official count day (e.g., 1 st day 
of testing or a semester or quarter). 


For evaluation purposes, the 
school student count is 
sometimes further prescribed 
with conditions such as the 
tally in tested grade levels or in 
grades one or higher. 


School Level 
Teacher Count 


The school level teacher count is a 
derived tally of core teachers based upon 
either the observed teacher count (via 
Spring survey of teachers at the middle 
school level or analysis of student course 
records at the secondary school level) or 
the Pupil Accounting Class report, which 
ever count is the larger. This method 
assures that the READ school number 
core teacher count reflects the individual 
classroom instructional situation 
employed by each school site. 


Deriving the teacher count for 
school from a variety of 
sources assures that the READ 
school number core teacher 
count reflects the individual 
classroom instructional 
situation employed by each 
school site. 


School Level 
Class Size 


The school level class size is a 
constructed class size statistic derived 
from the total number of students 
enrolled on the last day of the school 
year divided by the school teacher count. 


“School Teacher Count” is a 
READ-defined data element 
that represents the number of 
“classroom teachers” at a 
school site. Because “School 
Teacher Count” is used in the 
denominator, this constructed 
class size for each school is a 
robust estimate of each 
school’s average instructional 
unit, thus an improvement 
over the student-teacher ratio 
typically used in the literature 
(Ferguson, 1991). 
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Variable 


Definition 


Comment 


School Level 
Teacher College 
Training 


The school level teacher college 
training score is the mean of the 
academic training index of the core 
teacher belonging to the school site on 
the last day of the school year. Each core 
teacher is assigned an academic training 
index score on a seven point scale: 
l=Bachelors, 2=Bachelors+30 course 
credit hours (cch), 3=Masters/Equivalent, 
4=Masters+ 1 5(cch), 5=Masters+30(cch), 
6=Masters+60(cch), 7=Doctorate. 


Data source is Personnel 
Office college training code 
for individual teachers. 


School Level 
Teacher 
Certification 
Status 


The school level teacher certification 
status is the proportion of the core 
teaching belonging to the school site on 
the last day of the school year that have 
Standard Professional-I instructional 
certification, or higher. 


The school teacher 
certification status is derived 
as the mean of dichotomously 
coded teacher status: 
Provisional = 0, and 
Standard (or higher) = 1. 


School Level 
Teacher Service 
Years School 


The school level teacher experience at 
school site is the mean total number of 
years that the core teachers belonging to 
the school site on the last day of the 
school year were employed as teachers 
(provisional or certified) at the school 
site. 


Only the last continuous 
“belonging” period in the 
“event” school is considered. 
Data source is summarized 
Personnel Office records. An 
“event 7 school is location of 
interest to the evaluation study. 


School Level 
Teacher Service 
Years in the 
System 


The school level teacher experience in 
the system - is the mean total number of 
years that the core teachers belonging to 
the school site on the last day of the 
school year were employed as teachers 
(provisional or certified) in the system 
based upon their most recent occurrence 
of continuous employment in the system. 


Only the last continuous 
“belonging” period in the 
system is considered based on 
most recent hire date. Source 
is summarized Personnel 
Office data. 


School Level 
Teacher Service 
Experience 


The school level teacher years of 
experience is the recorded sum years of 
teaching credit assigned for teaching 
(provisional and certified) regardless of 
location. Does not have to be continuous 
years of teaching employment. 


Source is summarized 
Personnel Office data. 
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EXAMPLE RESULTS 



Table 2 presents descriptive analysis results from a previous 1994-95 effective schools 
study of 120 elementary schools in the Prince George’s County Public Schools. These results for 
16 variables include four of the school level input variables discussed in the previous data 
section: “Class Size,” “Teacher College Training,” “Teacher Service Years (at school site),” and 
“Teacher Service Years in PG System.” The results for these school level input variables were 
computed in conformance with the data specifications provided in Table 1. 

All reported results for the 16 variables listed in Table 1 are based on school-level 
aggregated values. For each variable considered, the minimum and maximum school-level 
values are reported in addition to the system average based on the unweighted school values. For 
example, the first listed variable, “class size”, indicates that there was one school with a mean 
teacher/student enrollment ratio of 16.8 to 1, that the school with the largest mean “class size” 
ratio was 31.9 to 1, and that the system average “class size” ratio was 25.6 to 1. Similar 
information is reported for all 16 variables listed. 
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Table 2 



Value-Added Assessment Study For 1994-95 School Year 

a 

Descriptive Analysis Results for Elementary School Level Variables 





N 


Minimum 


Maximum 


Mean 


Std. 

Deviation 


Class Size 


120 


16.8 


31.9 


25.58 


3.95 


Teacher Cost Per Student 


120 


1165 


2492 


1607 


271 


Enrollment MOBILITY: School 
(Days NOT enrolled in MSPAP 


116 b 


43.22 


277.20 


136.00 


38.11 


school for Past 3 years) 

Enrollment MOBILITY: System 
(Days NOT enrolled in PG system for 


120 


14.28 


119.56 


59.97 


21.36 


Past 3 yrs) 

% of MSPAP Examinees 
African-American 


120 


18.28 


100.00 


72.03 


22.47 


Mathematics MSPAP Scale Score 


120 


456.98 


551.70 


497.23 


19.47 


Reading MSPAP Scale Score 


120 


466.32 


529.73 


495.99 


12.76 


Science MSPAP Scale Score 


120 


456.94 


540.60 


497.50 


17.06 


% Poverty Among MSPAP 
Examinees 


120 • 


7.08 


96.00 • 


43.65 


20.55 


School Performance Index 


120 


22.3 


87.1 


45.45 


14.77 


% of MSPAP Examinees TAG 


120 


1.69 


75.47 


15.02 


11.70 


Teacher Days Absent in SY95 


120 


5.00 


22.39 


11.37 


2.93 


Teacher College Training (1-7 scale) 


120 


1.33 


3.59 


2.18 


.36 


Teacher Salary 


120 


31914 


51015 


40238 


3989 


Teacher Service Years at MSPAP 
School 


117 C 


2.38 


12.78 


6.97 


2.27 


Teacher Service Years in PG System 


120 


4.06 


22.00 


11.49 


3.90 


a - See school level variable definitions in Table 1. 

b- Four schools deleted from analysis because they provided less than three years of enrollment opportunity 





(i.e. t "combination" school with nonregular grade level structure or new schools). 
c ■ If schools were opened after the 1992-93 school year they were deleted from this computation due to 



truncated service opportunity. 
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Table 3 displays the overall summaries of the relationships between 14 of the school level 
variables from the 1994-95 effective schools study. These correlation results are presented as an 
example of the degree to which school input variables and school contextual variables are 
associated with student achievement when there is no statistical control or higher level modeling 
analysis of the variables considered. The point here is that simple, one-dimensional analysis 
results must be interpreted cautiously. For example, the correlation between class size and the 
schools’ average reading achievement is .27. This positive correlation indicates that larger 
classes tend to have higher reading achievement than smaller class sizes — contrary to prevailing 
beliefs. 



Although the relationships indicated in Table 3 are accurate they can be misleading or 
even counterintuitive when evaluating schools and setting school policy. For example, the 
positive correlation between class size and reading achievement does not take into account the 
fact that other variables (related to class size) may be causing the increased achievement rather 
than class size itself. For example, in Table 3 we see that larger classes have teachers with more 
years experience ( r = .38 for school and .42 for system years), higher levels of college training ( r 
= .19) and with larger numbers of talented and gifted (TAG) students ( r = .26). Because of the 
uncontrolled confounding influences of other variables present in these simple correlation 
results, concluding that higher class sizes improve student learning might be inaccurate. The 
point is that having “good data” is not enough, directors of evaluation must be careful to employ 
appropriate analysis methods that provide more resolution and filter out the influences of 
ancillary variables. 

Adcock and Sipes (1997) analyzed the 1994-95 data set with a multilevel design using 
HLM analysis, which simultaneously considers school-wide and individual student factors. This 
analysis revealed that Class Size was negatively related to Reading scale score. That is, the 
multilevel HLM analysis revealed that the relationship between class size and student 
achievement had actually changed directions (from a positive to a negative relationship) from 
that obtained with simple correlational analysis. The advantage of this multilevel design analysis 
is that much of the variability in the reading scores that is due to differences in students levels of 
poverty and student mobility can be removed through the HLM statistical analysis technology. 
This statistically significant relationship is consistent with prevailing literature that smaller class 
sizes benefit student performance. 
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Table 3 



Pearson Correlation Analysis Results For 
MS PAP Reading Performance 
And School Level Descriptive Variables 



Reading 
MSPAP 
Scale Scon 


9: Poverty 
Among 
MSPAP 
Examinees 


5* of 
MSPAP 
Examinees 
African- 
American 


Enrollment Enrollment 
Mobility: Mobility: 
School System 


Teacher 
Cost Per 
Student 


Teacher 
Days 
Absent in 
SY95 


"Class 

Size" 


Teacher 
Service 
Years at 
MSPAP 
School 


Teacher 
Service 
Years in PC 
System 


Teacher 

College 

Training 


Teacher 

Salary 


9c of 
MSPAP 
Examinees 
TAG 


School 

’erformance 

Index 


Reading 
MSPAP 
Scale Score 


1.00 


-.67** 


-.54** 


-.44** 


-.32** 


-.02 


.06 


.27** 


.29** 


.39** 


.40** 


.43** 


.52** 


.92** 


9c Poverty 
Among 
MSPAP 
Examinees 


-.67** 


1.00 


.47** 


.61** 


.42** 


.20* 


-.11 


-.41** 


-.30** 


-.37** 


-.29** 


-.39** 


-.50** 


-.58** 


9c of 

MSPAP 

Examinees 

African- 

American 


-.54** 


.47** 


1.00 


.40** 


.23* 


.33** 


-.05 


-.56** 


-.38** 


-.46** 


-.38** 


-.45** 


-.43** 


-.56** 


Enrollment 

Mobility: 

School 


-.44** 


.61** 


.40** 


1.00 


.70** 


-.01 


-.06 


-.24** 


-.30** 


-.40** 


-.37** 


-.45** 


-.37** 


-.40** 


Enrollment 

Mobility: 

System 


-.32** 


.42** 


.23* 


.70** 


1.00 


.04 


-.01 


-.17 


-.04 


-.25** 


-.22* 


-.26** 


-.39** 


-.29** 


Teacher 
Cost Per 
Student 


-.02 


.20* 


.33** 


-.01 


.04 


1.00 


.01 


-.82** 


.05 


.11 


.27** 


.21* 


-.16 


-.01 


Teacher 
Days 
Absent in 
SY95 


.06 


-.11 


-.05 


-.06 


-.01 


.01 


1.00 


.10 


.18 


.24** 


.17 


.21* 


.14 


.00 


"Class Size- 


.27** 


-.41** 


-.56** 


-.24** 


-.17 


-.82** 


.10 


1.00 


.38** 


.42** 


*.19* 


.37** 


' .26** 


.22* 


Teacher 
Service 
Years at 
MSPAP 
School 


.29** 


-.30** 


-.38** 


-.30** 


-.04 


.05 


.18 


.38** 


1.00 


.83** 


.51** 


.79** 


.09 


.23* 


Teacher 
Service 
Years in 
PG System 


.39** 


-.37** 


-.46** 


-.40** 


-.25** 


.11 


.24** 


.42** 


.83** 


1.00 


.68** 


.94** 


.19* 


.32** 


Teacher 

College 

Training 


.40** 


-.29** 


-.38** 


-.37** 


-.22* 


.27** 


.17 


.19* 


.51** 


.68** 


1.00 


.81** 


.25** 


.36** 


Teacher 

Salary 


.43** 


-.39** 


-.45** 


-.45** 


-.26** 


.21* 


.21* 


.37** 


.79** 


.94** 


.81** 


1.00 


.22* 


.37** 


% of 
MSPAP 
Examinees 
TAG 


.52** 


-.50** 


-.43** 


-.37** 


-.39** 


-.16 


.14 


.26** 


.09 


.19* 


.25** 


.22* 


1.00 


.58** 


School 

Performana 

Index 

(MSPP: 

SY95) 


.92** 


-.58** 


-.56** 


-.40** 


-.29** 


-.01 


.00 


.22* 


.23* 


.32** 


.36** 


.37** 


.58** 


1.00 



**. Correlation is significant at the 0.01 level (2-tailed), 
* Correlation is significant at the 0.05 level (2-tailed), 
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CONCLUDING REMARKS 



The school reconstitution challenge for school district evaluation directors is a function of 
two things: the question being asked and the nature of the recorded data. In the case of 
evaluating school reconstitution impact, this paper makes a case for broadening the question 
from student test gains” to an assessment of school input variables that are the targeted for 
reconstitution, i.e., teachers and class size. 

Specifying a more robust analysis model that includes school input variables, 
unfortunately, increases the challenge for evaluation directors to delineate, collect and prepare 
evaluation quality data for the variables in question. This paper provides detailed data 
specifications for populating a school reconstitution analysis model with student, teacher and 
school level variables. 

Finally, this paper provides example results of some of the school input variables of 
interest. The presented results revealed the need for directors of evaluation to use an appropriate 
level of analysis to guard against misinterpretation of findings. This paper introduced the topic 
of multilevel evaluation models which account for student, school contextual and school input 
variables in their natural settings. The authors hope to have communicated that no matter what 
the accountability demands are for reconstituted schools, the evaluation methods employed can 
not make up for faulty analysis model or bad data. 
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